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ABSTRACT 

The  UK  Ministry  of  Defence  (MOD)  has  a  vision  of  providing  Mission  Training  through  Distributed 
Simulation,  or  UK  MTDS.  The  paper  will  report  the  human  performance  results  from  a  three  year  MTDS 
Capability  Concept  Demonstrator  (CCD)  programme  presenting  data  to  qualify  the  development  of 
shared  situational  awareness  and  mental  models  in  a  networked  training  environment.  Evidence  was 
collected  during  five  national  and  multi-national  air  battlespace  exercises,  from  subjective  rating  scales, 
interviews,  observational  and  anecdotal  data. 

The  MTDS  CCD  facility,  based  at  RAF  Waddington,  included  8  fast-jet  Tornado  GR4  and  Typhoon 
simulators,  a  7-seat  AWACS  mission  crew  training  system,  and  extensive  exercise  management  capability 
(including  virtual  role  players  and  Computer  Generated  Forces  (CGF)).  Data  were  gathered  during 
representative  daily  Planning,  Briefing,  Execution,  and  Debriefing  (PBED)  cycles  of  air  battlespace 
missions  which  took  place  at  the  collective,  joint  and  multi-national  levels. 

The  MTDS  CCD  programme  assessed  the  training  benefit,  and  the  optimal  balance,  between  co-located 
teams  and  dispersed  training  teams  (both  in  the  UK  and  US).  The  major  source  of  data  across  all  sites 
was  the  application  of  the  Mission  Essential  Competency  (MEC)  framework.  Each  warfighter’s 
expectations  were  recorded  for  a  series  of  platform  based  MECs  and  compared  to  warfighter’s 
experiences.  These  data  were  used  to  assess  differences  between  role,  platform  and  site.  To  allow  analysis 
of  the  robustness  of  the  networked  environment  for  shared  cognitive  processes,  these  were  then  sub¬ 
divided  into  collective  experiences,  i.e.  those  that  required  interactions  with  other  players  or  entities 
leading  to  collective  training;  and  individual  MEC  experiences. 

Analysis  of  MEC  data  based  on  current  teamwork  theory  allowed  an  assessment  to  be  conducted  of  how 
successfully  networked  and  dispersed  training  audiences  were  being  supported  in  collective  teamwork. 
The  results  showed  that  all  warfighters  in  the  UK  and  US  received  significant  benefit  from  their 
participation  and  developed  shared  situational  awareness  enabling  them  to  successfully  meet  their 
training  objectives. 


INTRODUCTION 
Training  requirement 

The  UK  MTDS  programme  seeks  to  use  synthetic  training  environments  to  deliver  operational  team  and 
collective  training  for  the  air  component  of  the  Joint  Battlespace. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1 .  REPORT  DATE  2.  REPORT  TYPE 

OCT  2009  N/A 

4.  TITLE  AND  SUBTITLE 

The  Development  of  Shared  Mental  Models  during  Air  Battlespaee 
Collective  Training  using  Dispersed  Networked  Systems 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

QinetiQ  Cody  Technology  Park  Farnborough  Hants,  UK  GU14 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADA562561.  RTO-MP-HFM-181  Human  Performance  Enhancement  for  NATO  Military 
Operations  (Science,  Technology  and  Ethics)  (Amelioration  des  performances  humaines  dans  les 
operations  militaires  de  l’OTAN  (Science,  Technologie  et  Ethique)).  RTO  Human  Factors  and  Medicine 
Panel  (HFM)  Symposium  held  in  Sofia,  Bulgaria,  on  5-7  October  2009.,  The  original  document  contains 
color  images. 

14.  ABSTRACT 

The  UK  Ministry  of  Defence  (MOD)  has  a  vision  of  providing  Mission  Training  through  Distributed 
Simulation,  or  UK  MTDS.  The  paper  will  report  the  human  performance  results  from  a  three  year  MTDS 
Capability  Concept  Demonstrator  (CCD)  programme  presenting  data  to  qualify  the  development  of  shared 
situational  awareness  and  mental  models  in  a  networked  training  environment.  Evidence  was  collected 
during  five  national  and  multi-national  air  battlespaee  exercises,  from  subjective  rating  scales,  interviews, 
observational  and  anecdotal  data.  The  MTDS  CCD  facility,  based  at  RAF  Waddington,  included  8  fast-jet 
Tornado  GR4  and  Typhoon  simulators,  a  7-seat  AWACS  mission  crew  training  system,  and  extensive 
exercise  management  capability  (including  virtual  role  players  and  Computer  Generated  Forces  (CGF)). 
Data  were  gathered  during  representative  daily  Planning,  Briefing,  Execution,  and  Debriefing  (PBED) 
cycles  of  air  battlespaee  missions  which  took  place  at  the  collective,  joint  and  multi-national  levels.  The 
MTDS  CCD  programme  assessed  the  training  benefit,  and  the  optimal  balance,  between  co-located  teams 
and  dispersed  training  teams  (both  in  the  UK  and  US).  The  major  source  of  data  across  all  sites  was  the 
application  of  the  Mission  Essential  Competency  (MEC)  framework.  Each  warfighters  expectations  were 
recorded  for  a  series  of  platform  based  MECs  and  compared  to  warfighters  experiences.  These  data  were 
used  to  assess  differences  between  role,  platform  and  site.  To  allow  analysis  of  the  robustness  of  the 
networked  environment  for  shared  cognitive  processes,  these  were  then  sub-divided  into  collective 
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“Collective  training  involves  2  or  more  ‘teams’,  where  each  team  fulfils  different  ‘roles’,  training  to 
interoperate  in  an  environment  defined  by  a  common  set  of  collective  training  objectives” 

NATO  Study  Group  SAS-013 

The  MTDS  CCD  was  funded  to  de-risk  UK  MTDS  by  defining  the  user  requirements.  This  demonstrator 
programme  was  managed  by  UK  MOD  and  delivered  by  the  QinetiQ  led  Team  ACTIVE  (Aircrew 
Collective  Training  through  Immersive  Virtual  Events)  which  included  Boeing,  CueSim,  ATIL,  HVR  and 
Rockwell  Collins  between  2005  and  2008.  The  findings  of  the  programme  have  been  summarized  in  [1,  2, 
3].  The  output  of  the  programme  was  a  series  of  reports  based  on  studies  and  evidence  obtained  during  a 
programme  of  nine  exercises  conducted  using  Team  ACTIVE’S  facility  located  within  the  Air  Battlespace 
Training  Centre  (ABTC)  at  RAF  Waddington.  The  ABTC  was  linked  to  US,  UK  and  Canadian  distributed 
Mission  Training  Centres  (MTCs)  as  the  programme  progressed  as  indicated  in  Figure  1 . 

The  purpose  of  the  programme  was  to  address  a  number  of  Key  Investigative  Areas  (KLAs)  and  wider 
questions.  In  order  to  answer  these  KIAs,  UK,  US  and  Canadian  forces,  fast  jet,  rotary  wing,  Airborne 
Warning  and  Control  System  (AW ACS),  maritime  and  land  military  personnel,  participated  in  MTDS 
CCD  exercises  during  which  their  requirements  for  a  MTDS  capability  were  captured. 


Figure  1.  MTDS  CCD  dispersed  links 

When  considering  the  design  of  a  training  environment,  a  number  of  constituent  parts  must  be  considered 
which  together  deliver  an  overall  training  experience.  These  parts  included  the  delivery  of  an  exercise 
management  model,  the  provision  of  planning,  and  brief/debrief  tools  and  Synthetic  Training  Equipment 
(STE)  fidelity  requirements. 

While  much  of  the  work  conducted  before  the  MTDS  CCD  focused  on  the  collective  training  domain,  the 
MTDS  CCD  facility  provided  an  environment  to  examine  wider  operational  contexts,  including  joint 
collective  air  land  scenarios.  As  the  programme  evolved,  attention  widened  to  include  how  the  fast  jet 
crews  might  benefit  from  wider  training  with  Forward  Air  Controllers  (FACs),  AW  ACS,  rotary-wing 
aircraft  and  expanded  coalition  participation. 

An  assessment  of  the  impact  of  dispersion  on  training  benefit,  and  the  optimum  balance  between  co¬ 
located  and  dispersed  training  allows  us  to  test  assumptions  here  about  the  development  of  teamwork  and 
shared  mental  models. 
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MISSION  TRAINING  THROUGH  DISTRIBUTED  SIMULATION 
Training  Audience 

The  current  operational  tempo  and  the  predominance  of  coalition  operations  have  increased  the  need  for 
dispersed  training  and  training  across  national  borders  with  combined  partners.  Nine  events  were 
conducted  in  the  MTDS  CCD  facility,  five  of  which  involved  front  line  warfighters.  These  five  exercises 
were:  BATTLE  BUZZARD,  CONDOR  CAPTURE,  NORTHERN  GOSHAWK,  ARCTIC  OWL  and 
AVENGING  EAGLE.  Each  exercise  typically  lasted  a  week  with  a  familiarisation  day  and  then  a  build  up 
in  operational  complexity  over  the  next  3  or  4  days.  Typically,  the  fast  jet  warfighters  were  exposed  to  a 
range  of  mission  types  with  their  workload  increasing  over  the  exercise  week,  including  Close  Air  Support 
(CAS)  and  Air  Land  Integration  (ALI).  The  specific  participants  are  listed  in  Table  1  highlighting  the 
widening  range  of  potential  trainees  [4] . 


Roles  and  location  (ABTC  or  dispersed) 

UK  Tornado  GR4 

UK  Attack  Helicopter  (dispersed) 

UK  Typhoon 

UK  Type  42  Fighter  Controllers  (FCs)  (dispersed) 

UK  AW  ACS  (co-located  and  dispersed) 

UK  Intelligence  Officers 

UK  Air  Support  Operations  Centre  (ASOC) 

UK  Joint  Force  Air  Component  (JFAC)  mini  combat  ops 

US  F15c  Langley  (dispersed) 

UK  Forward  Air  Controller  (FAC) 

US  AFRL  F16  (dispersed) 

US  Distributed  Missions  Operation  Center  (DMOC)  F15e 
(dispersed) 

US  A10  Spangdahlem  (dispersed) 

US  DMOC  Control  and  Reporting  Centre  (CRC) 
(dispersed) 

US  A10  Mesa  (dispersed) 

US  DMOC  AW  ACS  (dispersed) 

US  AW  ACS  Tinker  (dispersed) 

US  DMOC  Virtual  Surveillance  Target  Attack  Radar 
Simulation  (VSTARS)  (dispersed) 

US  Joint  Terminal  Attack  Controller  (JTAC) 
(dispersed) 

Canadian  FAC  (dispersed) 

Table  1.  MTDS  CCD  training  participants. 


A  day  in  the  life  of  the  MTDS  CCD  facility  followed  the  cycle  of  PBED.  The  day  began  with  mass 
briefings  setting  the  context  and  overall  intent  for  the  day’s  mission.  Led  by  a  Mission  Commander,  the 
warfighters  would  then  begin  their  formation  planning,  and  coordinating  between  formations  (including 
across  national  boundaries),  as  required.  Following  the  completion  of  the  mission  execution  phase, 
warfighters  would  lead  a  series  of  debriefings,  beginning  with  a  formation  level  debriefing.  The  final 
activity  was  a  mass  debriefing  in  which  all  participants  ran  through  a  facilitated  After  Action  Review 
(AAR)  including  time  stamped  mission  replay,  to  capture  the  critical  lessons  identified. 
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TRAINING  ENVIRONMENT 

The  MTDS  CCD  facility,  illustrated  in  Figure  2,  consisted  of  fast  jet  simulators  (four  Typhoons  and  four 
Tornado  GR4  aircraft),  a  seven-seat  E-3  AW  ACS  capability,  and  a  comprehensive  exercise  management 
and  control  suite. 


Figure  2.  MTDS  CCD  components 


A  40-seat  briefing  and  de-briefing  room  and  a  selection  of  smaller  formation  planning  rooms  were 
provided.  These  incorporated  standard  in-service  planning  aids  and  video  conferencing,  telephone  and 
interactive  whiteboard  technology  so  that  warfighters  could  undertake  a  condensed  cycle  of  PBED.  A 
classified  networking  hub  connected  securely  with  training  facilities  in  the  UK,  US  and  elsewhere  in  the 
world  was  also  provided. 


SHARED  MENTAL  MODELS  AND  TEAMWORK 

“Teams  do  more  than  simply  interact  with  tools;  they  require  the  ability  to  coordinate  and  cooperatively 
interact  with  each  other  to  facilitate  task  objectives  though  a  shared  understanding  of  the  team’s  resources 
(e.g.,  members’  knowledge,  skills,  and  experiences),  the  team’s  goals  and  objectives,  and  the  constraints 
under  which  the  team  works.  Essentially,  teams  also  require  teamwork.” 

Salas,  Sims  and  Burke,  2005  [5] 

A  recent  review  of  teamwork  research  resulted  in  the  identification  of  the  “Big  Five”  components  of 
teamwork.  These  components  quoted  from  [p5,  p  560]  are: 

1.  Team  Leadership:  Ability  to  direct  and  coordinate  the  activities  of  other  team  members,  assess  team 
performance,  assign  tasks,  develop  team  knowledge,  skills,  and  abilities,  motivate  team  members,  plan  and 
organize,  and  establish  a  positive  atmosphere. 

2.  Mutual  performance  monitoring:  The  ability  to  develop  common  understandings  of  the  team 
environment  and  apply  appropriate  task  strategies  to  accurately  monitor  teammate  performance 
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3.  Backup  behaviour:  Ability  to  anticipate  other  team  members’  needs  through  accurate  knowledge  about 
their  responsibilities.  This  includes  the  ability  to  shift  workload  among  members  to  achieve  balance  during 
high  periods  of  workload  or  pressure. 

4.  Adaptability:  Ability  to  adjust  strategies  based  on  information  gathered  from  the  environment  through 
the  use  of  backup  behaviour  and  reallocation  of  intra-team  resources.  Altering  a  course  of  action  or  team 
repertoire  in  response  to  changing  conditions  (internal  or  external). 

5.  Team  orientation:  Propensity  to  take  other’s  behaviour  into  account  during  group  interaction  and  the 
belief  in  the  importance  of  team  goal’s  over  individual  members’  goals. 

In  support  of  these  are  three  mediating  functions: 

•  Shared  mental  models:  An  organizing  knowledge  structure  of  the  relationships  among  the  tasks  the 
team  is  engaged  in  and  how  the  team  members  will  interact. 

•  Mutual  trust:  The  shared  belief  that  team  members  will  perform  their  roles  and  protect  the  interests  of 
their  teammates. 

•  Closed-loop  communication:  The  exchange  of  information  between  a  sender  and  a  receiver 
irrespective  of  the  medium. 

Salas  et  al  comment,  “there  was  little  empirical  or  theoretical  research  available  to  guide  our 
understanding  of  how  the  “Big  Five”  and  the  coordinating  mechanisms  may  differ  as  teams  mature.”  [5]. 
They  report  that  team  leadership  and  orientation  will  be  significant  in  the  early  forming  of  the  team.  For 
the  newly  formed  distributed  and  co-located  teams,  clear  allocation  of  command  roles  will  be  important 
and  as  this  is  a  military  team,  leadership  will  be  conventionally  defined  rather  than  developed  ad  hoc. 

Team  orientation  is  more  challenging  for  the  dispersed  team  members  who  are  facing  more  formal  team 
forming  mechanisms.  During  the  MTDS  CCD  timeframe,  two  technology  enablers  were  introduced  to 
minimise  the  challenge:  shared  whiteboard  technologies  and  Video  Tele  Conference  (VTC)  links.  When 
planning,  team  leaders  at  all  locations,  had  as  much  access  to  these  tools  as  possible,  to  foster  team 
communication  and  development  [6].  Communication  during  this  team  development  time  has  often  been 
found  to  be  essential  for  good  team  performance  and  orientation. 

The  theoretical  propositions  and  elements  described  above  will  be  used  in  this  paper,  to  test  out  the 
validity  of  the  assumption  that  training  in  the  MTDS  environment  enabled  effective  collective  teamwork. 


ASSESSMENT  APPROACH 

Data  were  collected  by  UK  and  US  research  teams  at  each  event  using  a  range  of  objective  and  subjective 
data  collection  tools.  At  each  training  exercise  Team  ACTIVE  human  factors  experts  observed  groups  of 
the  UK  training  audience  and  the  exercise  management  team,  i.e.  the  white  force  and  role  players. 
Interviews  and  surveys  with  participants  in  the  UK  were  conducted.  These  were  supplemented  by  the  Air 
Force  Research  Laboratory  (AFRL)  Coalition  Mission  Training  Research  (CMTR)  toolset  [7,  8]  including 
both  UK  and  US  collected  data  collected  by  the  US  CMTR  research  team  at  all  sites. 


Tools 

To  understand  the  level  of  training  benefit,  a  number  of  surveys  and  human  factor  methods  were 
administered  to  co-located  and  dispersed  participants.  The  first  of  these  was  the  CMTR  Top  Three  Bottom 
Three  (T3B3)  survey,  distributed  to  the  training  audience,  role  players  and  white  force  every  day.  The  aim 
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was  to  obtain  the  top  and  bottom  three  outcomes  of  the  day.  Comments  were  also  used  to  support  the 
quantitative  outputs  from  other  data  analysis. 

Core  to  data  collection  were  MECs.  These  are  “Higher-order  individual,  team,  and  inter-team 
competencies  that  a  fully  prepared  pilot,  crew  or  flight  requires  for  successful  mission  completion  under 
adverse  conditions  and  in  a  non-permissive  environment”  [9,  10]. 

A  pre  and  post  MEC  survey  allowed  evaluation  of  the  difference  between  what  training  participants 
expected  to  receive  (CMTR  expectations  Survey  2a)  and,  after  the  exercise,  what  they  actually 
experienced  (CMTR  experiences  survey  2b).  These  CMTR  MEC  surveys  were  different  in  the  UK  and  US 
to  accommodate  for  differences  in  the  type  of  aircraft  and  terminology  used.  In  order  for  comparisons  to 
be  made  common  MECS  by  role  were  identified  [11], 

To  determine  if  warfighters  reported  that  the  training  had  value,  a  CMTR  Reactions  survey  was  used  to 
establish  how  the  aircrew  felt  about  the  MTDS  concept  as  compared  to  their  current  training. 


RESULTS 

In  order  to  assess  the  training  benefit  gained  by  co-located  and  dispersed  training  audiences,  a  comparison 
was  made  between  participants’  ratings  of  common  MECs  for  similar  roles,  such  as  air  to  air.  In  this  way, 
participants’  ratings  for  their  expectations  of  their  training  and  their  subsequent  experiences  could  be 
compared.  The  first  task  was  to  identify  MECs  that  were  common  between  the  ABTC  trainees  and  remote 
sites,  once  a  MEC  list  was  created  then  the  second  task  was  to  compare  these  ratings  statistically. 
Differences  between  sites  could  then  be  analysed  to  see  if  training  benefit  was  affected  by  dispersion. 

To  enable  a  comparison  between  the  UK  and  US,  the  CMTR  MECs  were  ‘cross-walked’.  This  means  that 
the  MECs  that  are  similar  or  equivalent  in  the  UK  and  US  surveys  have  been  combined  to  create  a  ‘cross- 
walked’  MEC. 

For  the  purposes  of  analysis  of  the  collective  performance,  these  MECs  were  then  divided  into  collective 
MECs,  i.e.  those  that  required  interactions  with  other  players  or  entities  leading  to  collective  training;  and 
individual  MECs. 

The  collective  MECs  were  compared  statistically  to  identify  significant  differences  between  co-located 
and  dispersed  STEs  in  the  UK  and  US.  From  these  differences,  inferences  can  be  drawn  about  the  impact 
of  dispersion  on  the  team  and  collective  training  experienced  at  each  site. 

These  collective  MEC  crosswalks  were  carried  out  on  the  air-to-air,  AW  ACS  and  air  to  ground  data 
collected  at  each  site.  For  the  purposes  of  this  paper,  the  detailed  analysis  of  the  latter  will  be  illustrated  in 
relation  to  teamwork  models.  Nonetheless,  analysis  of  the  other  two  major  roles  found  similar  effects. 

Key  results 

Over  the  course  of  the  MTDS  CCD,  a  significant  data  set  for  the  air  to  ground  role  was  collected  at  each 
participating  site.  From  the  data  gathered,  this  analysis  will  concentrate  on  data  relevant  to  our  validation 
of  the  teamwork  theory.  This  will  involve  the  reactions  that  aircrew  provided  to  the  exercises,  their  daily 
comments  and  a  comparison  of  collective  MECS. 

A  validation  of  previous  research  [12]  was  that  the  planning,  briefing  and  debriefing  (PBD)  activities  were 
as  important  in  development  of  teamwork  as  the  execution  phase.  Both  members  of  the  training  audience 
and  non-training  audience  commented  on  the  value  of  PBD  in  all  exercises.  Top  three  comments  from 
Tornado  GR4  aircrew  in  EXERCISE  AVENGING  EAGLE  include  “being  involved  in  the  brainstorm, 


15-6 


RTO-MP-HFM-1 81 


The  Development  of  Shared  Mental  Models  during 
Air  Battlespace  Collective  Training  using  Dispersed  Networked  Systems 


planning  and  debriefing”  demonstrating  the  utility  gained  from  this  process.  Comments  made  during  all 
the  exercises  show  that  both  Tornado  GR4  aircrew  and  AW  ACS  found  the  planning,  briefing  and 
debriefing  facilities  useful.  Typhoon  participants  also  commented  on  the  advantage  in  seeing  the  whole 
planning  cycle  and  that  the  experience  supported  planning  for  preparation  for  war: 

•  “The  opportunity  to  plan  an  unusual  scenario”  -  EXERCISE  NORTHERN  GOSHAWK 

•  “The  planning  co-ordination  with  US  forces  and  multiple  platforms  was  good  and  it  increased  my 
knowledge  of  how  they  operate  and  terms  they  use”  -  EXERCISE  AVENGING  EAGLE  Typhoon 

•  “the  mission  planning  was  enjoyable,  challenging  and  useful”  -  EXERCISE  CONDOR  CAPTURE 
Tornado  GR4 

Other  positive  T3B3  comments  on  the  debriefing  reinforced  the  need  for  focus  to  be  on  training  outcomes: 

•  “Bringing  in  aspects  of  the  mission  that  worked  well  into  the  debrief  provides  good  training”  - 
EXERCISE  ARCTIC  OWL  AW  ACS 

•  “Player  interaction  during  the  debrief’  -  EXERCISE  FALCON  FLIGHT  Tornado  GR4 

•  “A  frank  and  honest  debrief  including  how  we  can  improve  performance  -  EXERCISE  AVENGING 
EAGLE  AWACS 

•  “during  debrief  lots  of  talk  of  training  and  lessons  learned  and  little  of  tech  issues”  -  EXERCISE 
NORTHERN  GOSHAWK  Tornado  GR4 

•  “Mass  debrief  was  very  information  with  many  tactical  lessons  being  identified”  -  EXERCISE 
CONDOR  CAPTURE  Tornado  GR4 

During  the  mission  execution  phase,  major  findings  from  the  co-located  Tornado  GR4  crews  were  firstly, 
the  need  for  more  targeted  fidelity  and  currency  of  their  aircraft  simulation,  especially  the  range  of  sensor 
and  weapon  types.  Secondly,  for  greater  immersion  and  scene  detail,  the  warfighters  requested  greater 
horizontal  field  of  view  to  beyond  the  peripheral  vision  of  the  backseat  warfighter  [13]. 

This  paper  will  now  discuss  linking  these  results  to  an  analysis  of  teamwork  theory.  In  order  to  assess  how 
the  MTDS  programme  supported  teamwork  theory,  the  main  performance  indicators,  MECs,  were  rated 
according  to  the  importance  of  the  big  five  and  three  mediating  functions.  This  analysis  provides  us  with 
early  analysis  of  the  importance  of  training  objectives  at  the  team  level.  Prescriptively,  it  could  provide 
insight  for  white  force  and  trainees  of  the  key  aspects  that  must  be  incorporated  in  the  daily  plan  and 
reinforced  as  learning  points  in  the  AAR. 

For  the  collective  MECs,  Figure  3  provides  an  approximation  of  how  recent  team  theory  and  research 
could  be  used  to  balance  and  inform  training  design.  In  order  to  validate  the  percentages  provided,  it  is 
suggested  that  an  independent  rating  exercise  of  each  element  is  carried  out  by  a  team  of  role  specific 
warfighters. 
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Figure  3:  Illustration  of  how  an  assessment  of  teamwork  factors  could  be  used  to  inform  training 
design  and  delivery  based  on  the  model  described  in  Salas  et  al  (2005). 


Here  percentages  indicate  how  important  aspects  of  theory  might  be  for  the  roles  participating  in  MTDS. 
Further  postulation  on  the  link  between  team  theory  and  training  performance  assessment  would  suggest 
that  the  design  of  training  systems  could  be  validated  against  such  models.  For  example,  where  a  MEC 
rates  highly  on  many  individual  elements  and  is  considered  important  for  a  specific  training  need,  the 
design  of  the  training  environment  should  be  customised  to  meet  these  requirements.  Understanding  of  the 
training  need  in  terms  of  this  model  could  also  assist  the  training  instructors  in  designing  and  reinforcing 
key  teamwork  elements. 


Support  to  the  Big  five  as  illustrated  by  the  Air  to  Ground  participants 

The  data  were  reviewed  to  identify  evidence  for  the  five  teamwork  elements  and  mediating  functions 
defined  by  [5].  Evidence  from  subjective  ratings  by  the  Tornado  GR4  aircrew  indicates  that  training  was 
enhancing  their  shared  understanding  and  development  of  team  orientation.  The  crews  agreed  that: 

•  I  feel  I  will  be  better  prepared  to  lead  a  mission  due  to  this  training 

•  The  event  has  helped  me  to  improve  my  team  coordination  skills 

•  In  training  I  was  able  to  apply  lessons  learned  from  previous  missions  to  new  ones 
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In  terms  of  their  understanding  of  collective  team  training,  their  experience  of  ‘Operations  with  other 
formation/packages  (e.g.,  multi -package;  tanker,  AW  ACS;  air  to  ground,  air  to  air,  Composite  Air 
Operations  (COMAO)’  was  higher  than  their  expectations  providing  evidence  based  recommendations  for 
MTDS  utility.  Further  illustration  of  the  benefit  they  gained  from  working  with  other  teams  can  be  found 
in  some  examples  of  daily  T3B3  comments  during  EXERCISE  ARCTIC  OWL: 

•  “Mission  planning  good  to  see  more  COMAO/multiple  aircraft  integration” 

•  “Another  good  insight  into  COMAOs  -  especially  air  land  integration” 

•  “Good  coordination  work  between  ASOC/Unmanned  Air  Vehicle  (UAV)/Combined  Air  Operations 
Centre  (CAOC)/FAC  provides  good  effects  in  scenario” 

The  need  for  closed  loop  communications  were  reported  as  essential  to  build  the  air  picture  and  created 
immersion.  Interviews  in  EXERCISE  BATTLE  BUZZARD  established  that  the  radio  ‘chat’  is  “awesome” 
and  “top  training”  -  it  is  this  realistic  radio  traffic  that  provides  collective  training  benefit. 
Communications  with  other  US  players,  AW  ACS  and  FACs  were  seen  as  significant  contributors  to 
training.  “Good  comms  w/JTAC  -  critical  to  hitting  correct  tgt  [target]” 

Technical  problems  with  communications  were  often  reported  as  an  immersion  breaker,  but  when  working 
were  seen  as  a  critical  enabler  of  collective  training.  Such  an  effect  was  seen  with  all  players. 

Further,  consideration  of  the  MEC  survey  data  across  all  sites  using  crosswalks  allowed  many  aspects  of 
the  MTDS  CCD  to  be  qualified,  for  example,  the  fidelity  level  requirements  at  dispersed  and  co-located 
sites  [13]  and  the  range  of  training  feasible  [4]. 

In  this  analysis,  the  collective  MEC  data  using  survey  2a  and  2b,  was  analysed  in  relation  to  the  Big  Five 
[5].  A  subjective  rating  of  the  collective  MECs  provided  a  view  of  which  MECs  had  required  most,  if  not 
all,  of  the  elements  of  Salas  et  al’s  propositions  [5].  Overall,  analysis  of  the  collective  air  to  ground  MECs 
found  that  team  leadership  was  the  most  frequent  element  in  the  collective  MEC  set  (Figure  4).  Meeting 
this  need  in  the  MTDS  would  demonstrate  a  partial  fulfilment  of  predicted  team  development  [5] . 


21%  16% 


18% 


□  Shared  Mental  Models 

■  Mutual  Trust 

□  Closed  Loop  Comms 

□  Team  Leadership 

■  Mutual  Performance 
Monitoring 


Figure  4:  Approximate  distribution  of  teamwork  factors  in  the  air  to  ground  MECs 

Next,  the  verification  turned  to  an  analysis  of  the  success  with  which  the  MTDS  CCD  met  training 
experiences.  Table  2  highlights  several  collective  MECs  with  significant  elements  of  the  Big  5  and 
mediating  functions. 
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The  statistical  analysis  of  these  MECs  (expectations  vs.  experiences)  allows  a  qualification  of  how  well 
the  MTDS  programme  was  training  effective  team  and  hence  taskwork. 


MEC 

MTDS  findings 

Close  Air  Support  (CAS)  missions 

There  were  no  significant  differences  between  expectations  and 
experiences.  Trainees  were  receiving  the  level  of  training 

Multinational  and  joint  operations 

experience  that  they  had  anticipated  demonstrating  that  the  MTDS 
experience  was  meeting  their  expectations  whether  co-located  or 

Mission  planning 

dispersed.  This  infers  that  the  collective  teamwork  planned  and 
delivered  during  performance  met  their  needs. 

Suppression  of  Enemy  Air  Defence 
(SEAD)  training 

A  significant  interaction  was  found  between  experience, 
expectation  and  platform  (p<0.05).  The  co-located  Tornado  GR4s 
rated  their  experience  of  this  competency  significantly  lower  than 
expectation,  rating  it  as  not  at  all  effective.  This  is  likely  to  be  as  a 
result  of  simulator  fidelity  limitations.  The  dispersed  A  10s, 
however,  rated  their  experience  higher  than  expectation, 
suggesting  they  received  a  better  training  experience  than  the 
Tornado  GR4s.  It  is  interesting  to  note  that  dispersion  was  not  a 
disruptive  influence  on  the  teamwork  requirements. 

Combat  Search  And  Rescue 
(CSAR) 

A  trend  (P=0.06)  was  found  for  an  interaction  between  experience, 
expectation  and  platform.  As  per  the  previous  MEC,  the  Tornado 
GR4s  rated  their  training  experience  lower  than  expected,  and  the 
A  10s  ratings  were  higher  than  expectation.  For  the  co-located 
team,  this  was  reported  as  being  due  to  the  visual  field  of  view  and 
the  need  for  greater  visual  detail  at  low  altitude. 

EXERCISE  AVENGING  EAGLE  provided  the  most  effective 
training  experience,  rated  at  somewhat  effective,  with  the 
remaining  exercises  rated  at  slightly  effective.  This  MEC  was 
being  experienced  at  a  sufficient  level  but  should  be  simulated  to  a 
higher  degree  where  training  needs  require  it.  Of  interest  was  the 
rarity  of  training  in  this  MEC  and  that  it  was  welcomed  by 
warfighters  as  an  essential  and  unique  experience  in  synthetic 
environments.  Enhancements  to  enable  such  training  in  UK 
MTDS  in  simulation  are  ongoing  because  of  these  results. 

Problem  with  leadership  in  the  air 

A  significant  difference  (p<0.01)  was  found  with  overall 
experience  rated  lower  than  expectations.  There  was  a  significant 
difference  between  exercises  (p<0.05).  EXERCISE  AVENGING 
EAGLE  was  the  most  effective  at  providing  training  experience  of 
leadership  fallout,  rated  at  somewhat  effective,  probably  as  a  result 
of  the  complexity  of  participants  and  a  strong  blue  planning 
process.  It  is  likely  that  by  this  final  exercise,  both  the 
technologies  and  management  of  training  were  more  capable  of 
generating  realistic  experiences  [6]. 
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MEC 

MTDS  findings 

COMAO  operations  and  operations 
with  other  formations  and  packages 

Overall,  the  data  show  that  aircrew  received  significantly  (p<0.05) 
more  effective  training  experience  than  expected.  Experience 
ratings  were  highest  in  EXERCISE  ARCTIC  OWL,  rated  at  very 
effective.  This  exercise  provided  a  novel  interaction  of  air,  land 
and  maritime  assets  in  a  complex  synthetic  battlespace. 

The  overall  average  rating  across  all  exercises  was  quite  effective, 
which  is  akin  to  aircrews’  rating  of  this  MEC  experience  being 
met  to  a  very  good  extent.  The  MTDS  CCD  was  then  meeting  the 
collective  training  need. 

Push  CAS/package  fallout 

Overall,  experience  was  rated  significantly  lower  than  expectation 
(p<0.001).  However,  in  EXERCISE  CONDOR  CAPTURE  the 
training  experience  was  rated  significantly  higher  than 
expectations,  being  rated  at  quite  effective.  Package  fallout  can  be 
interlinked  with  unexpected  leadership  handover  and  so  more 
training  opportunity  is  needed  to  enhance  learning. 

Table  2:  Examples  of  Air  to  Ground  collective  MECS  with  significant  Big  5  elements 


The  foregoing  analysis  highlights  the  interaction  between  the  training  design  and  environment  with  team 
outcomes.  It  is  worth  noting  that  in  spite  of  specific  equipment  fidelity  issues,  air  to  ground  training  was 
successful  at  all  sites  delivering  collective  team  of  team  training.  Future  enhancements  are  already 
ongoing  to  target  the  mission  execution  capability  enabling  deeper  experience  of  specific  MECs.  This  will 
allow  consideration  of  team  as  well  as  inter -team  training.  As  the  dispersed  A- 10  results  illustrate, 
collective  teamwork  can  be  delivered  at  each  site  given  a  sufficient  level  of  equipment  fidelity.  More  so, 
manipulation  of  the  aspects  of  the  Big  5,  forcing  the  team  to  work  together  through  planned  injects, 
resulted  in  stronger  learning.  Communications  and  joint  planning  were  particularly  significant  as  an 
enabler  of  collective  performance. 


DISCUSSION 

The  MTDS  CCD  aim  was  to  provide  warfighters  with  an  environment  in  which  they  could  learn  to  operate 
at  a  team  of  teams’  level  with  national  and  coalition  partners.  For  this  to  occur  successfully  there  needed 
to  be  sufficiency  in  the  training  environment  through  PBED.  Communication  between  dispersed  teams  via 
shared  VTCs  and  open  platform  communications  during  planned  events  supported  the  development  and 
delivery  of  concerted  actions. 

Safeguards  between  sites,  both  in  terms  of  technologies,  exercise  management  and  shared  data,  maximised 
the  opportunity  to  work  as  a  collaborative  team  [6].  Of  significant  value  was  the  facilitation  of  the 
communications  between  the  warfighters  and  role  players  so  that  they  were  sufficiently  realistic  to 
immerse  and  hence  train  participants.  For  example,  in  communications  with  other  US  players,  AW  ACS 
and  FACs  were  seen  as  significant  contributors  to  training,  e.g.  “Realistic  comms  with  other  real  assets  - 
just  like  real  life”  (AW ACS  warfighter).  Finally,  decision  making  under  pressure  where  warfighters  had  a 
limited  time  to  act/react  to  situations  and  coordinate  with  others  was  experienced  successfully.  These 
results  will  be  described  in  detail,  and  presented  next  in  the  context  of  current  teamwork  theory. 

Leadership  of  the  collective  team  was  directed  by  overall  directed  by  Rules  of  Engagements  (ROE)  and 
translated  by  the  air  commander  into  an  overall  mission  plan,  subsequently  delivered  by  a  series  of  sub 
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teams.  Actions  by  adversary  forces  pushed  these  team  and  leadership  skills  to  their  limits  to  exercise 
collective  performance.  It  was  essential  then  that  teams  had  sufficient  situational  awareness  and  shared 
mental  models  trusting  each  other  to  deliver  the  required  effect;  and  to  respond  flexibly  to  changes 
enforced  by  the  adversaries.  These  included  needing  to  adapt  to  damage/loss  of  allies,  the  ability  to  replan 
and  deal  with  new  targets/orders  in  the  rapidly  changing  environment,  all  while  maintaining  clear  lines  of 
command  and  communications  throughout. 

Hence,  the  MTDS  environment  can  be  analysed  by  the  teamwork  model  described  in  [5]  providing  a  case 
study  to  test  out  some  of  the  predictions  made.  Of  the  fundamentals  of  teamwork,  the  Big  Five,  how  well 
did  the  MTDS  experience  support  their  development? 

As  the  programme  and  capability  is  still  maturing  some  aspects  of  the  MTDS  CCD  findings  are  relevant 
only  to  the  facilities  of  that  time  and  so  reassessment  at  a  later  stage  may  be  necessary.  For  example,  the 
co-located  air  to  ground  cockpits  need  more  targeted  fidelity  so  that  wider  and  more  realistic  missions  can 
occur,  e.g.  in  the  range  of  weaponry  and  sensor  suite.  The  MTDS  CCD  capability  is  undergoing  upgrades 
to  add  detailed  targetted  fidelity  to  the  facility  in  response  to  these  findings.  Despite  these  limitations  for 
intra-team  operations,  when  participants  were  questioned  over  inter-team  operational  utility,  they  were 
very  positive  about  the  collective  training  benefit. 

Other  teamwork  aspects  supported  the  vision  of  MTDS  as  an  environment  for  team  and  individual  skills 
training.  Specifically,  being  in  command  in  the  air  of  a  large  COMAO  gave  package  leaders  an  experience 
rare  in  the  real  world.  The  ability  to  lead  a  team  was  welcomed  both  by  experienced  crew  and  by  those 
more  inexperienced  as  essential  practice.  It  allowed  rehearsal  of  leadership  and  decision  making  due  to 
manipulations  by  the  red  force,  and  general  ‘mission  friction’.  Also,  critical  to  the  delivery  of  the  required 
effect  was  the  ability  to  monitor  each  other’s  performance.  Lessons  were  learnt  in  collaborative  AAR, 
where  each  site  linked  together  in  VTCs  and  a  timed  replay  of  critical  events  occurred.  Shared  mental 
models  were  facilitated  by  such  joint  planning  and  review  and  further  enhanced  by  active  learning  during 
VTC  brainstorms,  e.g.  “The  planning  co-ordination  with  US  forces  and  multiple  platforms  was  good  and  it 
increased  my  knowledge  of  how  they  operate  and  terms  they  use”  -  RAF  Typhoon  pilot. 

In  common  with  the  need  to  adapt  during  the  planning  phase,  an  expert  white  force  assured  that  the  level 
and  tempo  was  appropriately  managed.  They  facilitated  learning  during  mission  execution  by  moderating 
the  ability  and  actions  of  the  red  force,  changing  the  ROE,  prioritising  team  training  objectives  and 
inserting  events  and  intelligence  through  role  players.  Mutual  trust  developed  between  co-located  and 
dispersed  sites  during  the  event  week  so  that  new  requirements  were  met  through  pre-defined  roles  and 
plans. 

During  PBED,  closed  loop  communications  were  a  critical  enabler  of  backup  behaviour,  the  importance  of 
comms  across  the  teams  reinforced  learning  experienced  across  roles  (e.g.  air  land  integration)  and 
cultures  (e.g.  learning  the  process  and  language  differences,  especially  terminology).  These  elements 
further  enforced  the  development  of  shared  mental  models.  Moreover,  where  comms  performance  was 
missing  or  unpredictable,  participants  reported  that  they  felt  it  gave  them  a  more  realistic  experience  of 
current  ops  but  others  were  less  favourable,  reporting  that  any  loss  of  comms  prevented  team  learning. 

The  final  aspect  required  for  an  effective  team,  is  team  orientation  “the  belief  in  the  importance  of  team 
goal’s  over  individual  members’  Goals”  [5].  Examples  of  this  were  evident  in  mission  execution  and 
during  AARs;  discussions  were  led  by  the  white  force  highlighting  how  specific  missions  required  an 
effect  to  be  delivered  that  was  more  important  than  a  team  goal.  For  example,  when  a  ground  commander 
called  for  air  support  of  troops  under  fire  and  the  air  assets  were  heavily  involved  in  completing  their  own 
plan.  In  this  case,  the  fast  jet  team’s  response  to  the  ground  requirement  was  untimely  and  the  AAR 
reinforced  the  bigger  picture  of  the  air  land  battlespace  and  how  supporting  own  troops  at  that  point  was 
more  significant  that  delivery  of  individual  effects.  Such  team  learning  is  vital  for  current  operations  and 
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as  a  result  of  the  MTDS  CCD  programme,  the  capability  is  being  used  for  Distributed  Synthetic  Air  Land 
Training  (DSALT)  furthering  the  understanding  between  the  air  and  land  element  such  as  in  current 
operational  theatres. 


CONCLUSIONS 

The  MTDS  CCD  programme  was  successful  in  providing  collective  training  for  the  co-located  and 
dispersed  participants  showing  an  increasing  degree  of  relevant  operational  utility  as  the  facilities  and 
management  of  dispersion  matured.  It  coped  with  a  widening  training  audience  by  implementing 
safeguards  and  analysis  of  trainee  performance  pre  and  post  each  exercise  using  the  MEC  approach. 

Analysis  of  the  training  team  data  in  relation  to  the  teamwork  theory  helps  us  understand  how  this 
networked  collective  environment  enabled  appropriate  teamwork  that  resulted  in  acceptable  mission 
outcomes.  These  findings  provide  further  validation  of  MTDS  within  and  across  teams  and  national 
boundaries. 

ACKNOWLEDGEMENTS 

The  author  wishes  to  thank  her  colleagues  from  Team  ACTIVE,  in  particular  Will  Clark,  Sanjay  Khetia 
and  the  human  factors  team  and  exercise  leaders:  Dr  Elizabeth  S  Gehr,  James  Kearse  and  Dr  Dale 
Richards.  I  am  grateful  also  for  the  provision  of  the  AFRL  MEC  data  through  multinational  collaboration 
between  the  UK  MoD  and  US  facilitated  by  Dr  Wink  Bennett,  US  AFRL.  Finally,  to  all  participants  for 
their  contributions  throughout. 

REFERENCES 

[1]  Saltmarsh,  J.  and  MacKenzie,  S  (2008).  The  Future  of  Collective  Training:  Mission  Training  through 
Distributed  Simulation.  RUSI  Defence  Systems. 

[2]  Dudfield,  H.J.,  Beattie,  D.,  and  Clark,  W.  (2008)  MTDS  Final  Report.  QinetiQ  Customer  Report. 

[3]  Costello,  Wg.  Cdr.  (2009)  Distributed  Training  special  paper.  In  Proceedings  of  SimTecT,  Adelaide, 
Australia. 

[4]  Kearse,  J.,  Dudfield,  H.J.,  Gehr,  S.E.  and  Clark  (2009)  Lessons  Learnt  In  The  Range  Of  Team 
Training  Experienced  In  The  UK  Mission  Training  Through  Distributed  Simulation  Demonstrator 
Programme.  In  Proceedings  of  SimTecT,  Adelaide,  Australia. 

[5]  Salas,  E.,  D.E.  Sims,  and  C.S.  Burke,  Is  there  a  "Big  Five"  in  teamwork?  Small  Group  Research, 
2005.  36(5):  p.  555-599. 

[6]  Dudfield,  H.J.  and  Kearse,  J  (in  prep)  Managing  distributed  training  events  -  lessons  learned  from 
the  Mission  Training  through  Distributed  Simulation  Capability  Concept  Demonstrator.  In 
Proceedings  of  ITT  SEC  Orlando  2009. 

[7]  Gehr,  S.E.,  Schurig,  M.A.,  Jacobs,  L.,  van  der  Pal,  J.,  Smith,  E.,  McIntyre,  H.,  Gehr,  S.E.,  Schurig, 
M.,  Symons,  S.,  Schreiber,  B.  and  Bennett  Jr.,  W.  (2005)  Evaluating  the  Impacts  of  Mission  Training 
via  Distributed  Simulation  on  Live  Exercise  Performance:  Results  from  the  US/UK  “Red  Skies” 
Study.  In  The  Effectiveness  of  Modelling  and  Simulation  -  From  Anecdotal  to  Substantive  Evidence. 
Meeting  Proceedings  RTO-MP-MSG-035. 


RTO-MP-HFM-181 


15-13 


The  Development  of  Shared  Mental  Models  during 

Air  Battlespace  Collective  Training  using  Dispersed  Networked  Systems 


e-tar'. 


[8]  Bennett,  W.,  Jr.,  and  Schreiber,  B.S.  (2005).  Assessing  the  training  potential  of  MTDS  in  exercise 
First  WAVE.  Paper  presented  at  symposium  on  the  Effectiveness  of  Modeling  and  Simulation  -  from 
Anecdotal  to  Substantive  Evidence,  Warsaw,  Poland 

[9]  Colegrove,  C.  and  Bennett,  W.,  Jr.  (2005).  Competency-based  training:  Adapting  to  warfighter 
needs.  Presentation  at  the  RAeS  Conference:  Multi  Role  and  Networked  Simulation,  London,  UK. 

[10]  Colegrove,  C.  M.  and  Alliger,  G.  M.  (2002).  Mission  Essential  Competencies:  Defining  Combat 
Mission  Readiness  in  a  Novel  Way.  Paper  presented  at  the  NATO  RTO  Studies,  Analysis  and 
Simulation  Panel  (SAS)  Symposium.  Brussels,  Belgium. 

[11]  Dudfield,  H.J.,  Kearse,  J,  McAuliffe,  C.,  Watson,  S.,  Gehr,  S.E  and  Clark,  W.  The  Impact  Of  Fidelity 
Levels  On  Distributed  And  Co-located  Fast  Jet  Participants  During  Multinational  Collective 
Training.  In  Proceedings  of  SimTecT,  Adelaide,  Australia. 

[12]  Smith,  E.  (2003)  Report  on  UK  MTDS  Study,  Unpublished  MOD  report. 

[13]  Dudfield,  H.J.  and  Kearse,  J  (in  prep)  Assessing  Needs  of  the  Warfighter  in  Distributed  Collective 
Training  Simulations.  In  Proceedings  ofllTSEC  Orlando  2009. 


15  - 14 


RTO-MP-HFM-1 81 


